Trainable, Scalable Summarization Using Robust NLP and Machine Learning

We describe a trainable and scalable summarization system which utilizes features derived from information retrieval, inibrmation extraction, and NLP techniques and on-line resources. The system con> bines these features using a trainable feature combiner learned from summary examples through a machine learning algorithm. We demonstrate system scalability by reporting results on the best combination of summarizat ion features for different document sources. We also present preliminary results from a task-based evaluation on summarization outpnt usability. 1 I n t r o d u c t i o n Frequency-based (Edmundson, 196(.); Kupiec, Pedersen, and Chen, 1995; Brandow, Mitze. and Rau, 1995), knowledge-based (Reimer and Hahn, 1988; McKeown and Radev, 1995), and discoursebased (Johnson et al., 1993; Miike et al., 1994; Jones, 1995) approaches to automated summarization correspond to a continuum of increasing understanding of the text and increasing complexity in text processing. Given the goal of machine-generated summaries, these approaches a t tempt to answer three central questions: • How does the system count words to calculate worthiness for summarization? • How does the system incorporate the knowledge of the domain represented in tile text? • How does the system create a coherent and cohesive summary? Our work leverages off of research in these three approaches and a t tempts to remedy some of the difficulties encountered in each by applying a combination of information retrieval, information extraction, *We would like to thank Ja.mie Callan for his help with the INQUERY experiments. and NLP techniques and on-line resources with nmchine learning to generate summaries. Our DimSum system follows a common paradigm of sentence extraction, but automates acquiring candidate knowledge and learns what knowledge is necessary to sun> inarize. We present how we automatically acquire caudidate features in Section 2. Section 3 describes our training methodology for combining features to generate summaries, and discusses evaluation results of both batch and machine learning methods. Section 4 reports our task-based evalnation. 2 E x t r a c t i n g F e a t u r e s Ill this section, we describe how the system counts linguistically-motivated, autornaticallyderived words and nmlti-words in calculating worthiness for smnmarizat.ion. We show how tile systetll uses an external corpus t.o incorporate domain knowledge in contrast to text-only statistics. Finally, we explain how we a t tempt to increase the co hesiveness of our summaries by using name aliasing, WordNet synonyms, and morphological variants. 2.1 D e f i n i n g Single a n d M u l t i w o r d T e r m s Frequency-based summarizat ion systems typically use a single word string as the unit for counting fl'equency. Though robust, such a method ignores the semantic content of words and their potential men> bership in multi-word phrases and may introduce noise in frequency counting by treating the same strings uniformly regardless of context. Our approach, similar to (Tzoukerman, Klavans, and aacquemin, 1997), is to apply NLP tools to extract multi-word phrases automatically with high accuracy and use them as the basic unit in the summarization process, including frequency calculation. Our system uses both text statistics (term frequency, or /.at) and corpus statistics (inverse document frequency, or idJ) (Salton and McGill, 1983) to derive sigTzal~zrc words as one of the sunmlarization features. If single words were the sole basis of counting for our summarizat ion application, noise would be